Image capture apparatus and control method thereof capable of reducing an image data amount

ABSTRACT

Disclosed are an image processing apparatus that can efficiently suppress the data amount of data indicating a spatial distribution and an angular distribution of light intensity and a control method of the same. The image processing apparatus obtains data indicating the spatial distribution and the angular distribution of the intensity of light beams that have passed through partial pupil areas obtained by dividing the exit pupil of an imaging optical system into a predetermined number. The image processing apparatus then reduces the bit depth or the number of tones of signals constituting the data based on the predetermined number.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of application Ser. No. 15/272,654,filed Sep. 22, 2016 the entire disclosure of which is herebyincorporated by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image processing apparatus and animage processing method, and in particular relates to a technique forreducing an image data amount. The present invention also relates to animage capture apparatus and a control method thereof.

Description of the Related Art

There are known image capture apparatuses that divide the exit pupil ofan imaging lens (imaging optical system) into a plurality of pupilareas, and can generate a plurality of images each corresponding to onepupil area with one shooting operation (U.S. Pat. No. 4,410,804). Theseplurality of images have a parallax therebetween (parallax images), andare thus information indicating the spatial distribution and angulardistribution of light intensity, and have properties similar to those ofdata called light field (LF) data.

Ren. Ng, et al., “Light Field Photography with a Hand-Held PlenopticCamera” (Stanford Tech Report CTSR 2005-02, 2005 Apr. 20) discloses arefocusing technique for changing an in-focus area of a captured imageafter shooting, by using LF data to reconstruct an image in an imagingplane (virtual imaging plane) that is different from the imaging planeat the time of shooting.

In the case of a configuration in which a pupil area is divided using animage sensor in which the photoelectric conversion area of one pixel isdivided into a plurality of (n) portions, a maximum of n parallax imagesare generated in one shooting operation. Therefore, the data amount ofthe image is n times an image that is generated in the case where thepupil area is not divided (normal image). Furthermore, if the parallaximages are combined and an image corresponding to a normal image isadded in consideration of compatibility with a device that cannot handleparallax images, the data amount will be n+1 times the normal image.

SUMMARY OF THE INVENTION

The present invention has been made in light of the issue of suchconventional techniques, and provides an image processing apparatus andan image processing method that can efficiently suppress the data amountof data indicating the spatial distribution and angular distribution oflight intensity.

According to an aspect of the present invention, there is provided animage processing apparatus comprising: an obtaining unit configured toobtain data that is based on light beams that have passed throughpartial pupil areas obtained by dividing an exit pupil of an imagingoptical system into a predetermined number N_(p); and a reduction unitconfigured to reduce a bit depth or a number of tones of signalsconstituting the data based on the predetermined number N_(p).

According to another aspect of the present invention, there is providedan image capture apparatus comprising: an image sensor that generatesdata indicating a spatial distribution and an angular distribution of anintensity of light beams that have passed through partial pupil areasobtained by dividing an exit pupil of an imaging optical system into apredetermined number N_(p); and an image processing apparatuscomprising: an obtaining unit configured to obtain the data generated bythe image sensor; and a reduction unit configured to reduce a bit depthor a number of tones of signals constituting the data based on thepredetermined number N_(p).

According to another aspect of the present invention, there is providedan image capture apparatus, comprising: an image sensor in which aplurality of imaging pixels are arranged, each of the plurality ofimaging pixels including a plurality of subpixels, and each of theplurality of subpixels receiving a light beam that has passed through adifferent one of partial pupil areas obtained by dividing an exit pupilof an imaging optical system into a predetermined number N_(p); and ageneration unit configured to generate captured image data obtained bycombining signals of the plurality of subpixels for each of the imagingpixels, and light field data constituted by the signals of the pluralityof subpixels, wherein when generating the light field data, thegeneration unit determines a bit depth or a number of tones of thesignals of the plurality of subpixels based on the predetermined numberN_(p) so as to be smaller than that of pixel signals of the capturedimage data.

According to another aspect of the present invention, there is providedan image processing method executed by an image processing apparatus,the method comprising: obtaining data that is based on light beams thathave passed through partial pupil areas obtained by dividing an exitpupil of an imaging optical system into a predetermined number N_(p);and reducing a bit depth or a number of tones of signals constitutingthe data based on the predetermined number N_(p).

According to still another aspect of the present invention, there isprovided a control method of an image capture apparatus having an imagesensor in which a plurality of imaging pixels are arranged, wherein eachof the plurality of imaging pixels includes a plurality of subpixels,and each of the plurality of subpixels receives a light beam that haspassed through a different one of partial pupil areas obtained bydividing an exit pupil of an imaging optical system into a predeterminednumber N_(p), the control method comprising: generating, in accordancewith a mode, captured image data obtained by combining signals of theplurality of subpixels for each of the imaging pixels, and light fielddata constituted by the signals of the plurality of subpixels, wherein,in the generating, when generating the light field data, a bit depth ora number of tones of the signals of the plurality of subpixels isdetermined based on the predetermined number N_(p) so as to be smallerthan that of pixel signals of the captured image data.

According to further aspect of the present invention, there is provideda non-transitory computer-readable storage medium storing a program forcausing a computer to function as an image processing apparatuscomprising: an obtaining unit configured to obtain data that is based onlight beams that have passed through partial pupil areas obtained bydividing an exit pupil of an imaging optical system into a predeterminednumber N_(p); and a reduction unit configured to reduce a bit depth or anumber of tones of signals constituting the data based on thepredetermined number N_(p).

According to still further aspect of the present invention, there isprovided a non-transitory computer-readable storage medium storing aprogram for causing a computer of an image capture apparatus comprisesan image sensor in which a plurality of imaging pixels are arranged,each of the plurality of imaging pixels including a plurality ofsubpixels, and each of the plurality of subpixels receiving a light beamthat has passed through a different one of partial pupil areas obtainedby dividing an exit pupil of an imaging optical system into apredetermined number N_(p), to function as a generation unit configuredto generate captured image data obtained by combining signals of theplurality of subpixels for each of the imaging pixels, and light fielddata constituted by the signals of the plurality of subpixels, whereinwhen generating the light field data, the generation unit determines abit depth or a number of tones of the signals of the plurality ofsubpixels based on the predetermined number N_(p) so as to be smallerthan that of pixel signals of the captured image data.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing an example of a function configuration of adigital still camera serving as an example of an image processingapparatus according to embodiments of the present invention.

FIG. 2 is a schematic diagram of pixel arrangement in the embodiments ofthe present invention.

FIGS. 3A and 3B are respectively a schematic plan view and a schematiccross-sectional view of a pixel in the embodiments of the presentinvention.

FIGS. 4A and 4B are schematic explanatory diagrams of a pixel structurethat is optically roughly equivalent to a pixel structure in theembodiments of the present invention.

FIG. 5 is a schematic explanatory diagram of a pixel and pupil divisionin the embodiments of the present invention.

FIGS. 6A to 6C are diagrams respectively illustrating a relationshipbetween an image sensor and pupil division, a relationship between adefocus amount and an image shift amount between parallax images, and arange in which refocusing is possible, according to the embodiments ofthe present invention.

FIG. 7 is a schematic diagram showing a relationship between subpixelsand angle information that can be obtained, according to the embodimentsof the present invention.

FIG. 8 is a schematic explanatory diagram of refocusing processing inthe embodiments of the present invention.

FIGS. 9A and 9B are flowcharts related to LF data reduction processingin the embodiments of the present invention.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will now be described indetail in accordance with the accompanying drawings. Note that in theembodiments that will be described below, a configuration will bedescribed in which the present invention is applied to an image captureapparatus as an example of an image processing apparatus, but theconfiguration related to image capture is not essential to the presentinvention. For example, a configuration may be adopted in which lightfield data that has already been recorded as an image file in a certainformat is obtained from a storage apparatus or an external apparatus.Light field data (LF data) is an example of data indicating the spatialdistribution and angular distribution of the light intensity.

Overall Configuration

FIG. 1 is a diagram showing an example of the function configuration ofa digital still camera 100 (hereinafter, simply referred to as thecamera 100) as an example of the image capture apparatus according tothe embodiments of the present invention.

A first lens group 101 is arranged at the front of an imaging opticalsystem, and is held so as to be movable back and forth along the opticalaxis. A shutter 102 functions not only as a shutter for controlling anexposure duration at the time of capturing still images, but also as anaperture for adjusting the light amount at the time of image capture byadjusting the opening diameter. A second lens group 103 arrangedrearward of the shutter 102 (image sensor side) can move back and forthalong the optical axis integrally with the shutter 102, and realizes azoom function together with the first lens group 101.

A third lens group 105 is a focus lens, and can move back and forthalong the optical axis. The first to third lens groups 101, 103 and 105and the shutter 102 constitute the imaging optical system. An opticallow pass filter 106 is arranged in front of an image sensor 107, andreduces false colors and moire that occur in captured images. The imagesensor 107 is constituted by a two-dimensional CMOS image sensor andperipheral circuitries. In this embodiment, the image sensor 107 is atwo-dimensional single-plate color image sensor in which a plurality oflight receiving elements (pixels), namely, m (>1) light receivingelements in the horizontal direction and n (>1) light receiving elementsin the vertical direction, are two dimensionally arranged, and primarycolor mosaic filters in a Bayer array are formed on the light receivingelements. The color filters restrict the wavelength of transmissionlight that is incident on the light receiving elements in units ofpixels.

A zoom actuator 111 rotates a cam barrel (not illustrated) so as todrive at least one of the first lens group 101 and the second lens group103 along the optical axis in accordance with control by a zoom drivingcircuit 129, and realizes a zoom function. A shutter actuator 112controls the opening diameter of the shutter 102 so as to adjust animaging light amount, and controls an exposure duration at the time ofcapturing a still image, in accordance with control by a shutter drivingcircuit 128.

A focus actuator 114 drives the third lens group 105 along the opticalaxis in accordance with control by a focus driving circuit 126.

A flash 115 is preferably a flash lighting apparatus that uses a xenontube, but may be an illumination apparatus provided with an LED thatcontinuously emits light. An AF auxiliary light output unit 116 projectsan image of a mask having a predetermined opening pattern via a lightprojecting lens, and improves focus detection ability with respect to anobject having a low luminance and an object having a low contrast.

A CPU 121 controls the overall operations of the camera 100, and has acalculation unit, a ROM, a RAM, an A/D converter, a D/A converter, acommunication interface circuit and the like (not illustrated). The CPU121 executes a program stored in the ROM so as to control variouscircuits of the camera 100 and realizes the functions of the camera 100such as AF, AE, image processing and recording.

A flash control circuit 122 controls lighting of the flash 115 insynchronization with an image capture operation. An auxiliary lightcircuit 123 controls the lighting of the AF auxiliary light output unit116 at the time of a focus detection operation. An image sensor drivingcircuit 124 controls the operations of the image sensor 107, and alsoA/D converts image signals read out from the image sensor 107 andoutputs the image signals to the CPU 121. An image processing circuit125 applies image processing such as gamma conversion, colorinterpolation, scaling and JPEG encoding/decoding to image signals.

The focus driving circuit 126 moves the third lens group 105 along theoptical axis by driving the focus actuator 114 based on a focusdetection result, and performs focus adjustment. The shutter drivingcircuit 128 controls the opening diameter and open/close timing of theshutter 102 by driving the shutter actuator 112. The zoom drivingcircuit 129 drives the zoom actuator 111 in accordance with a zoomoperation that is input by an image taker pressing a zoom operationswitch included in operation switches 132, for example.

A display device 131 is an LCD or the like, and displays informationregarding an image capture mode of the camera 100, preview images beforeimage capture and images for checking after image capture, informationregarding an in-focus state at the time of focus detection, and thelike. The operation switches 132 include a power switch, a release(image capture trigger) switch, the zoom operation switch, an imagecapture mode selection switch and the like. A recording medium 133 is aremovable semiconductor memory card, for example, and records capturedimages.

Image Sensor

FIG. 2 is a diagram schematically showing an arrangement example ofimaging pixels and subpixels in the image sensor 107, andrepresentatively shows an area in which the imaging pixels are arrangedin an array of 4 horizontal pixels×4 vertical pixels. In thisembodiment, the photoelectric conversion area of each imaging pixel isdivided into four in the vertical direction and four in the horizontaldirection, and each of the divided photoelectric conversion areasfunctions as a subpixel. Therefore, FIG. 2 can be said to show an areain which the subpixels are arranged in an array of 16 horizontalsubpixels×16 vertical subpixels. As will be described later, in theimage sensor 107 of this embodiment, a plurality of imaging pixels eachhaving a plurality of subpixels are arranged, and the subpixels arrangedin each imaging pixel individually receive light beams that have passedthrough mutually different exit partial pupil areas (partial pupilareas) of the imaging optical system.

In this embodiment, a pixel group 200 having 2×2 pixels at the upperleft in FIG. 2 corresponds to the unit of repetition of primary colorfilters in a Bayer array provided in the image sensor 107. Accordingly,a pixel 200R having an R (red) spectral sensitivity is arranged at theupper left, pixels 200G having a G (green) spectral sensitivity arearranged at the upper right and lower left, and a pixel 200B having a B(blue) spectral sensitivity is arranged at the lower right. Also, asrepresentatively indicated by the pixel at the upper right in FIG. 2, ineach imaging pixel, N_(θ)×N_(θ) (here, N_(θ)=4) subpixels 201 to 216 aretwo-dimensionally arranged.

By arranging a large number of the 4×4 imaging pixel (16×16 subpixel)array shown in FIG. 2 in the image capture plane of the image sensor107, a plurality of parallax images (light field data) can be obtained.In addition, subpixels also function as focus detection pixels, and thusfocus detection that employs an image capture plane phase differencedetection method using various positions of the screen as focusdetection areas can be performed. In this embodiment, it is assumed thatthe imaging pixel pitch (period) ΔX is 9.2 μm both horizontally andvertically, and the number of effective pixels N_(LF) is 3900 columnshorizontally×2600 rows vertically=10.14 mega pixels. Also, it is assumedthat the subpixel period Δx is 2.3 μm both horizontally and vertically,and the number of effective subpixels N is 15600 columnshorizontally×10400 rows vertically=approximately 1.62 giga subpixels.

FIG. 3A shows a plan view of one of the imaging pixels shown in FIG. 2(here, the pixel 200G is used, but the pixels 200B and 200R have thesame configuration) when viewed from a light receiving surface side (+Zside) of the image sensor, and FIG. 3B shows a cross-sectional view ofthe a-a cross-section in FIG. 3A when viewed from the −y side.

As shown in FIGS. 3A and 3B, in the pixel 200G of this embodiment, amicrolens 350 for collecting incident light is formed on thelight-receiving side of the pixel, and photoelectric conversion portions301 to 316 obtained by dividing the pixel into N_(θ) (four) in the xdirection and N_(θ) (four) in the y direction are formed. Thephotoelectric conversion portions 301 to 316 respectively correspond tothe subpixels 201 to 216. Note that the microlens 350 is formed into ashape formed by N_(θ) (four) sub microlenses 351 to 354 being in linecontact with adjacent sub microlenses. The optical axes (apexes) of thesub microlenses 351 to 354 are eccentric from the centers of areasobtained by dividing the pixel area into 2×2=4, toward the center of thepixel area. A broken line 370 in FIG. 3B indicates the optical axes(apexes) of the sub microlenses 353 and 354.

The photoelectric conversion portions 301 to 316 may be a pin structurephotodiode having an intrinsic layer sandwiched between a p-type layerand an n-type layer, or may be a pn junction photodiode with theintrinsic layer omitted as necessary.

In each pixel, a color filter 360 is formed between the microlens 350and the photoelectric conversion portions 301 to 316. As necessary, thespectral transmittance of the color filter may be changed for eachsubpixel, or the color filter may be omitted.

Light that is incident to the pixel 200G shown in FIGS. 3A and 3B iscollected by the microlens 350, separated by the color filter 360, andafter that, received by the photoelectric conversion portions 301 to316.

In the photoelectric conversion portions 301 to 316, after electron-holepairs are generated in accordance with the amount of the received light,and separated by a depletion layer, the electrons having a negativeelectric charge are accumulated on the n-type layer, and the holes aredischarged outside of the image sensor 107 via a p-type layer 300connected to a constant voltage source (not illustrated).

An electrostatic capacity portion (floating diffusion: FD) 320 and atransfer gate 330 are formed adjacent to each of the photoelectricconversion portions 301 to 316. Furthermore, wiring layers 340 that alsofunction as light-shielding layers are formed between the microlens 350and the electrostatic capacity portion (FD) 320. The electrostaticcapacity portion (FD) 320 is arranged in an area in which the lightcollected by the microlens 350 is not incident.

The electrons accumulated on the n-type layers of the photoelectricconversion portions 301 to 316 are transferred to the electrostaticcapacity portion (FD) 320 via the transfer gates 330, and are convertedinto voltage signals. Note that the voltage signals that underwentconversion in the electrostatic capacity portion (FD) 320 are output tocolumn circuits provided in the image sensor 107 for the respectivecolumns. The column circuits each include a capacity element forsampling the voltage signals, an amplifier for amplifying the signalamplitude, an A/D converter for converting the voltage signals intodigital signals, and the like. Note that in addition to these, a memoryfor storing the digital signals may be provided in each of the columncircuits. Moreover, in the column circuit of the image sensor 107 inthis embodiment, a single slope type A/D converter for comparing avoltage signal value with a reference voltage (ramp signals) thatchanges over time is provided as an A/D converter.

Furthermore, the electrons accumulated in the photoelectric conversionportions 301 to 316 can be individually read out, but can also be addedtogether in predetermined units and read out. Units for readout includea unit of two horizontal pixels and two vertical pixels, a unit of fourpixels only in the vertical or horizontal position direction and thelike. In this case, the addition may be performed by the electrostaticcapacity portion (FD) 320 or each of the column circuits.

FIG. 4A shows a schematic cross-sectional view and a schematic plan viewof the pixel shown in FIGS. 3A and 3B. Also, FIG. 4B shows a schematiccross-sectional view and a schematic plan view of a pixel opticallyroughly equivalent to the pixel shown in FIGS. 3A and 3B. In the pixelshown in FIG. 4A, if reconstruction is performed such that all of theoptical axes (apexes) 370 of the sub microlenses 351 to 354 thatconstitute the microlens 350 overlap, the configuration in FIG. 4B isobtained. By constituting the microlens 350 with the four submicrolenses 351 to 354, it is possible to optically suppress theinfluence from the separation bands between the photoelectric conversionportions 306, 307, 310 and 311 that are near the center of the pixelarea, the electrostatic capacity portion (FD) 320, and the regions ofthe wiring layers 340 also serving as light-shielding layers.

FIG. 5 schematically shows pupil division of the photoelectricconversion portion in the pixel structure shown in FIG. 4B. In FIG. 5,the x-axis and y-axis in the cross-sectional view are inversed fromthose in FIGS. 3A and 3B and FIGS. 4A to 4B so as to correspond to thecoordinate axes of the exit pupil plane.

The image sensor 107 is arranged near the imaging plane of the imagingoptical system, and light beams from the object pass through an exitpupil 400 of the imaging optical system, and are incident to individualphotoelectric conversion areas (subpixels). Partial pupil areas 501 to516 have a substantially conjugate relationship with the light receivingsurfaces of the photoelectric conversion portions 301 to 316 (thesubpixels 201 to 216) obtained by N_(θ)×N_(θ) (4×4) division, due to themicrolens 350. Therefore, the partial pupil areas 501 to 516 representpartial pupil areas in which the individual photoelectric conversionportions (subpixels) can receive light. Moreover, a pupil area 500 is apupil area in which light can be received by the entire pixel 200G thatincludes the photoelectric conversion portions 301 to 316 (the subpixels201 to 216) divided into N_(θ)×N_(θ) (4×4).

The pupil distance from the imaging plane to the exit pupil plane isseveral tens of millimeters, while the diameter of the microlens 350 isseveral micrometers, and thus the aperture value of the microlens 350 istens of thousands, thereby causing diffraction blur at the level ofseveral tens of millimeters. Therefore, the images on the lightreceiving surfaces of the photoelectric conversion portions 301 to 316have the pupil intensity distribution (the incident angle distributionof the light receiving rate), rather than becoming distinct pupil areasor partial pupil areas.

FIG. 6A shows a schematic diagram showing the correspondence relationbetween the image sensor and the pupil division according to thisembodiment. Here, the partial pupil areas 509 to 512 arerepresentatively shown. Light beams that passed through the partialpupil areas 509 to 512 are incident to the subpixels 209 to 212 of theimage sensor at angles different from each other and is received at thephotoelectric conversion portions 309 to 312, respectively. Similarly,light beams that passed through the partial pupil areas 513 to 516 arereceived at the photoelectric conversion portions 313 to 316, lightbeams that passed through the partial pupil areas 505 to 508 is receivedat the photoelectric conversion portions 305 to 308, and light beamsthat passed through the partial pupil areas 501 to 504 is received atthe photoelectric conversion portions 301 to 304.

Therefore, the photoelectric conversion portions 301 to 316 (thesubpixels 201 to 216) that share the microlens 350 receive light beamsthat passed through mutually different partial pupil areas. The outputread from the subpixels 201 to 216 is LF data indicating the spatialdistribution and angular distribution of the light intensity.

A parallax image corresponding to one specific partial pupil area amongthe partial pupil areas 501 to 516 of the imaging optical system can beobtained by collecting one corresponding signal out of those of thesubpixels 201 to 216 from the data of a plurality of imaging pixels thatconstitute the LF data. For example, if the signal of the subpixel 209(the photoelectric conversion portion 309) is collected from the data ofthe imaging pixels, it is possible to obtain a parallax image thatcorresponds to the partial pupil area 509 of the imaging optical systemand has a resolution formed of the number of effective pixels. The samecan be applied to the other subpixels. Therefore, in this embodiment, aplurality of parallax images (of which number is equal to the number ofpartial pupil areas N_(p)=N_(θ)×N_(θ)), corresponding to the respectivepartial pupil areas can be obtained by an image sensor in which aplurality of pixels each having a plurality of subpixels for receivinglight beams that pass through mutually different partial pupil areas arearrayed.

Moreover, it is possible to generate a captured image having aresolution of the number of effective pixels, by adding (combining) allthe signals of the subpixels 201 to 216 for each of the imaging pixels.

Relationship Between Defocus Amount and Image Shift Amount

The relationship between a defocus amount and an image shift amount ofLF data (parallax image group) that can be obtained by the image sensor107 of this embodiment will be described below.

FIG. 6B shows a schematic diagram showing the relationship between adefocus amount and an image shift amount between parallax images. Theimage sensor is arranged in an image capture plane 800, and similarly toFIGS. 5 and 6A, the exit pupil of the imaging optical system is dividedinto N_(p) areas, namely, the partial pupil areas 501 to 516 (here,divided into 16 areas).

A magnitude |d| of a defocus amount d is a distance from the imageforming position of the object to the image capture plane 800. A case inwhich the defocus amount d is negative (d<0) refers to a front focusingstate in which the image forming position of the object is on the objectside relative to the image capture plane 800, and a case in which thedefocus amount d is positive (d>0) refers to a rear focusing state inwhich the image forming position of the object is on the opposite sideto the object relative to the image capture plane 800. In an in-focusstate in which the image forming position of the object is in the imagecapture plane 800, the magnitude of the defocus amount d is 0. FIG. 6Bshows an example in which an object 801 is in an in-focus state (d=0),and an object 802 is in a front focusing state (d<0). The front focusingstate (d<0) and the rear focusing state (d>0) are collectively called adefocused state (|d|>0).

In a front focusing state (d<0), out of light beams from the object 802,light beams that have passed through the partial pupil areas 509 to 512are collected at a position on the object side relative to the imagecapture plane 800. After that, the light beams widen to widths Γ09 toΓ12 centered on centroid positions G09 to G12 of the light beams, andforms blurred images on the image capture plane 800. The blurred imagesare received by the subpixels 209 to 212 constituting the imaging pixelsarranged in the image sensor, and parallax images are generated.Therefore, in the image constituted by the signals of the subpixels 209of the imaging pixels, an image of the object 802 that is blurred to awidth Γ09 is recorded at the centroid position G09. In imagesconstituted by the respective subpixels 210 to 212, images of the object802 that are blurred to widths Γ10 to Γ12 are recorded at the centroidpositions G10 to G12. Note that the same applies to light beams thathave passed through the partial pupil areas 513 to 516, 505 to 508, and501 to 504.

Blurring widths Γ (Γ01 to Γ16) of the object images increasesubstantially in proportion with the increase in the magnitude |d| ofthe defocus amount d. Similarly, a magnitude |p| of an image shiftamount p between object images of parallax images also increasessubstantially in proportion with the increase in the magnitude |d| ofthe defocus amount d. The image shift amount p is a difference betweencentroid positions of the light beams, and, for example, the magnitudeof the image shift amount between an image constituted by the output ofthe subpixels 209 and an image constituted by the output of thesubpixels 212 is |G09-G12|.

Also in the rear focusing state (d>0), the same applies except that theobject image deviation direction between the parallax images is oppositeto that in the front focusing state. In the in-focus state (the defocusamount d=0), the centroid positions of the object images of the parallaximages match, and image deviation does not occur (the image shift amountp=0).

Therefore, as the magnitude |d| of the defocus amount d increases, themagnitude of the image shift amount between a plurality of parallaximages constituting LF data increases.

In this embodiment, it is possible to perform focus detection by animage capture plane phase difference method, by calculating an imageshift amount between parallax images by a correlation operation, usingthe relationship in which as the magnitude |d| of the defocus amount dincreases, the magnitude of the image shift amount between parallaximages increases. Note that if necessary, focus detection may beperformed using a focus detection apparatus that is configuredseparately from the image sensor 107 and employs a phase differencesystem, or focus detection by a contrast method may be performed usingparallax images or captured images.

Refocusable Range

Next, refocusing processing and a refocusable range will be described.First, angle information of incident light beams that can be obtained bysubpixels will be described with reference to FIG. 7. Here, a subpixelperiod is denoted by Δx, the number of subpixels for each pixel isdenoted by N_(p)=N_(θ)×N_(θ), and a pixel period is denoted byΔX=N_(θ)Δx. Also, Δθ=Θ/N_(θ), where Δθ is the angular resolution, and Θis the estimated angle of the exit pupil of the imaging optical system.If paraxial approximation is used, a relational expression N_(θ)F≈1/Δθsubstantially holds true, where F is an aperture value of the imagingoptical system. Out of all the light beams that are incident on thepixels, light beams at incident angles θ₀ to θ₃ are respectivelyincident on the subpixels 212 to 209 each at the width of the angularresolution A.

FIG. 8 shows a schematic explanatory view of refocusing processing inthis embodiment. FIG. 8 schematically shows, using line segments, pixelsX_(i) (i=0 to N_(LF)−1) of the image sensor that are arranged in theimage capture plane. Light beams that are incident on an i-th pixelX_(i) at an angle θ_(a) (a=0 to N_(θ)−1) are received by each of thesubpixels. The signals of the subpixels that received the light beamsare denoted by L_(i, a) (a=0 to N_(θ)−1).

An image (refocused image) in the virtual image capture plane whoseposition on the optical axis is different from that in the image captureplane in which the image sensor is arranged can be generated from LFdata obtained by shooting. Processing for generating a refocused imagefrom LF data is called refocusing processing. A captured image and arefocused image generated from LF data are collectively calledreconstructed images, and processing for generating a reconstructedimage from LF data is called reconstruction processing.

A position at which light beams intersect the virtual image captureplane after being incident on the subpixels in each pixel in therefocused image capture plane and travelling in this state is determinedfrom a distance d′ between the image capture plane and the virtual imagecapture plane, and the incidence direction (the angle θ_(a)) of thelight beams. Therefore, an imaging signal obtained from an imaging pixelof the virtual imaging plane can be obtained by combining the signals ofthe subpixels that received, in the image capture plane, light beamsthat are incident on the pixel position. By generating, from LF data,the imaging signals of all the imaging pixels of the virtual imagingplane in this manner, a refocused image corresponding to the virtualimaging plane can be obtained. In actuality, the refocused image can beobtained by processing for shifting the positions of the parallax imagesand performing weighted addition. Here, coefficients used for theweighted addition form a group of coefficients whose values are allpositive, and the sum of which is 1.

FIG. 6C shows a schematic explanatory view of a range in whichrefocusing is possible according to this embodiment. When a permissiblecircle of confusion is δ, and an aperture value of the imaging opticalsystem is F, a depth of field at the aperture value F is ±Fδ. Incontrast, regarding an effective aperture value F₀₉ (F₀₁ to F₁₆) ofpartial pupil areas that underwent N_(θ)×N_(θ) division and becamesmaller, F₀₉=N_(θ)F holds true. Therefore, the effective depth of fieldof a parallax image is ±N_(θ)Fδ, which is N_(θ) times deeper than in acase where pupil division is not performed, and the in-focus rangewidens N_(θ) times. Accordingly, the LF data obtained by shooting at theaperture value F of the imaging optical system is constituted by a groupof parallax images that have the effective depth of field ±N_(θ)Fδ.Therefore, it is possible to obtain a refocused image that is in-focusin the virtual imaging plane at any distance in the range of theeffective depth of field ±N_(θ)Fδ. This corresponds to readjusting(refocusing), after shooting, the in-focus range of the captured image(the distance of the object that is in focus). Therefore, the effectivedepth of field ±N_(θ)Fδ is a range in which refocusing is possible.Regarding the outside of the range in which refocusing is possible, onlya blurred object image is present in the LF data, and thus it is notpossible to generate a refocused image corresponding to the virtualimage capture plane positioned out of the range in which refocusing ispossible (the focal distance cannot be changed to a distance outside therange in which refocusing is possible).

It can be said that a range in which refocusing is possible is a defocusamount during shooting that can be eliminated after shooting. Therefore,if the magnitude |d| of defocus during shooting is substantially in arange that satisfies Expression (1) below, it is possible to performfocusing by performing refocusing.|d|≤N _(θ)Fδ  (1)

Note that the size of the permissible circle of confusion δ isdetermined by δ=2ΔX (the inverse number of a Nyquist frequency 1/(2ΔX)of the pixel period ΔX) or the like.

Reduction in LF Data Amount

As mentioned above, in the case where the number of divided pupil areasN_(p)=N_(θ)×N_(θ), if each of the subpixel signals that constitute LFdata has the same bit depth or the same number of tones as normalcaptured image data, the data amount will be N_(p) times the amount ofthe normal captured image data.

However, the signal of each of the pixels of a reconstructed image (acaptured image or a refocused image) is generated by combining thesubpixel signals for the number of divided pupil areas N_(p). Combiningsubpixel signals for generating one pixel signal is weighted additionthat uses coefficients whose sum is 1. Therefore, assuming that a bitdepth necessary for the reconstructed image is a first number of bits b1(b1 bits/pixel), if subpixel signals are generated at a bit depth of asecond number of bits b2, which is expressed byb2≥b1−(log₂ N _(p))  (2),the bit depth (the number of tones) necessary for the reconstructedimage can be maintained. Moreover, if b2<b1, the LF data amount can bereduced.

In Expression (2), if the first and second numbers of bits b1 and b2 arerespectively replaced with the numbers of tones t1 (=b1²) and t2 (=b2²),t2≥t1/N _(p)  (3).

In this case as well, if the number of tones t2<the number of tones t1,the LF data amount can be reduced. Note that the pupil division numberN_(p) used in Expressions (2) and (3) above does not necessarily need tomatch the number of the photoelectric conversion portions included inthe unit pixel of the image sensor 107. For example, in the case ofadding and reading a plurality of subpixel signals, the number ofsubpixel signals per unit pixel after the addition will be the number ofdivided pupil areas or partial pupil areas N_(p). In other words, areduction amount of LF data amount is desirably variable in accordancewith the number of added subpixel signals.

Note that in the case of generating a reconstructed image from subpixelsignals whose data amount has been reduced in this manner, weightingcoefficients used for the combining are changed in accordance with thereduction in data amount. For example, a ratio of weighting coefficientsin the case where the data amount is not reduced to an average value(1/N_(p)) can be used. Note that this is an example, and the weightingcoefficients may be changed by other methods.

A configuration may be adopted in which the camera 100 of thisembodiment includes a first record mode in which a captured image isrecorded as normal image data, and a second record mode in which acaptured image is recorded as LF data, and one of the modes can beselected via an operation of the operation switch 132, for example. Asshown in FIG. 9A, for example, when generating recording data, the CPUchecks whether the first record mode or the second record mode isselected (step S901). In the case where the first record mode isselected, the CPU 121 combines, in units of imaging pixels, subpixelsignals read out via the image sensor driving circuit 124, and generatescaptured image data having a predetermined bit depth (the first numberof bits) or a predetermined number of tones (the first number of tones)(step S902). In the case where the second record mode is selected, theCPU 121 generates light field data in which the subpixel signals readout via the image sensor driving circuit 124 are indicated by apredetermined bit depth (the second number of bits) or a predeterminednumber of tones (the second number of tones) (step S903). Note that theCPU 121 does not need to perform all of the conversions of the bit depthor the number of tones. A hardware circuit for the conversions or thelike may be provided.

When the number of subpixels (the number of divided exit pupils)obtained by dividing the exit pupil of the imaging optical system isN_(p), by setting the second bit number to an integer that is greaterthan or equal to the first number of bits−(log₂N_(p)), the toneproperties of a reconstructed image generated from LF data can be madegreater than or equivalent to those of the captured image data.Moreover, the LF data amount can be reduced by setting the second numberof bits to less than the first number of bits.

Alternatively, by making the second number of tones greater than orequal to the first number of tones/the number of divided pupil areas(N_(p)), the tone properties of the reconstructed image generated fromthe LF data can be made greater than or equivalent to those of thecaptured image data. Also, by making the second number of tones smallerthan the first number of tones, the LF data amount can be reduced.

The captured image data or LF data that has been generated may berecorded in a predetermined file format in the recording medium 133, forexample. Note that the bit depth or the number of tones of the LF datacan be recorded as header information of the file. Accordingly, in anapparatus that uses LF data, a reconstructed image can be correctlygenerated.

Note that in an example shown in FIG. 9A, the conversion of the bitdepth or the number of tones is performed by the CPU 121 as a part ofoperations for generating recording data. However, the conversion of thebit depth or the number of tones can be performed in another operations.For example, the conversion may be performed in operations forgenerating digital signals in the A/D converters in the image sensor107. More specifically, the bit depth or the number of tones can becontrolled in accordance with an operational mode so as to be in therange indicated by Expression (2) or (3), when voltage signals, whichare analog signals, are input to the A/D converter provided for each ofthe columns of the image sensor 107. Operations of the image sensor 107,such as a readout mode and the like are controlled by the image sensordriving circuit 124, and thus the operations of the image sensor 107 canbe easily changed based on the number of divided pupil areas N_(p).Furthermore, the A/D converters provided in the image sensor 107 aresingle slope type A/D converters, and thus there is also an effect ofenabling the conversion time to be reduced by reducing the bit depth orthe number of tones.

The camera of this embodiment has an image sensor in which a pluralityof imaging pixels each having a plurality of subpixels are arranged, andeach of the plurality of subpixels receives a light beam that has passedthrough a different one of a plurality of partial pupil areas that wereobtained by dividing the exit pupil of the imaging optical system intopredetermined numbers. Moreover, the camera includes a first record modein which imaging signals obtained by combining signals of a plurality ofsubpixels for each imaging pixel are recorded as captured image data,and a second record mode in which signals of a plurality of subpixelsare recorded as LF data without being combined. The camera sets, in thesecond record mode, the bit depth or the number of tones of the subpixelsignals to less than the bit depth or the number of tones of the imagingsignals that are recorded in the first record mode. Therefore, the LFdata amount can be reduced by more than in the case of recording withthe same bit depth or number of tones as the captured image data.Furthermore, by determining the bit depth or the number of tones in thesecond record mode by taking the number of divided pupil areas intoconsideration, it is possible to record the LF data that enablesgeneration of a reconstructed image having a bit depth or the number oftones that is greater than or equivalent to that of the captured imagedata that is recorded in the first record mode. Note that in the case ofreducing the bit depth or the number of tones, the reduction amount maybe determined in accordance with the bit depth (for example, 8 bits and14 bits) of the recording format that is used when recording the data tothe recording medium 133, or the number of display tones of the displaydevice 131.

Other Embodiment

In the above-described embodiment, a case was described in which LF dataindicating the spatial distribution and angular distribution of theintensity of light beams that have passed through partial pupil areasobtained by dividing the exit pupil of the imaging optical system into apredetermined numbers is generated through reading out from the imagesensor. However, recorded LF data can also be used. For example, if thebit depth b1 or the number of tones t1 of recorded LF data is reduced,the LF data amount can be reduced.

In this case, for example, as shown in FIG. 9B, the CPU 121 obtains thenumber of divided pupil areas or the number of subpixels correspondingto the LF data from header information of an LF data file, for example(step S911). At this time, the bit depth or the number of tonesnecessary for a reconstructed image may also be obtained, if necessaryor possible to obtain.

The CPU 121 then reduces the bit depth or the number of tones of signals(subpixel signals) constituting LF data that has not been reduced, to avalue determined based on the number of pupil division or the number ofdivided pupil areas (step S913). Specifically, the CPU 121 reduces thenumber of bits before the reduction to a value (>0) that is less thanlog₂N_(p), or the number of tones before the reduction to a number thatis greater than or equal to 1/(the number of the divided pupil areas)and less than the number of tones before the reduction.

If the bit depth or the number of tones necessary for a reconstructedimage has been obtained in step S911, the bit depth or the number oftones after the reduction can be determined so as to be smaller than theobtained bit depth or number of tones.

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2015-194398, filed on Sep. 30, 2015, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image capture apparatus, comprising: an imagesensor in which a plurality of imaging pixels are arranged, each of theplurality of imaging pixels including a plurality of subpixels, and eachof the plurality of subpixels receiving a light beam that has passedthrough a different one of partial pupil areas obtained by dividing anexit pupil of an imaging optical system into a predetermined numberN_(p), wherein N_(p) is an integer greater than 1; and a processor and amemory which function as: a generation unit configured to generatecaptured image data by combining signals of the plurality of subpixelsfor each of the plurality of imaging pixels, and light field dataconstituted by the signals of the plurality of subpixels withoutcombining the signals of the plurality of subpixels, wherein whengenerating the light field data, the generation unit determines a numberof tones of the signals of the plurality of subpixels based on thepredetermined number N_(p) so as to be smaller than a number of tones ofthe captured image data.
 2. The image capture apparatus according toclaim 1, wherein in a case where a bit depth of the pixel signals of thecaptured image data is b1 bits and a bit depth of the signals of theplurality of subpixels is b2 bits, the generation unit determines thebit depth b2 so as to satisfy b1>b2≥b1−(log₂N_(p)), wherein b1, b2, eachis an integer greater than
 1. 3. The image capture apparatus accordingto claim 1, wherein in a case where the number of tones of the pixelsignals of the captured image data is t1 and the number of tones of thesignals of the plurality of subpixels is t2, the generation unitdetermines the number of tones b2 so as to satisfy t1>t2≥b2/N_(p),wherein t1, t2, b2, each is an integer greater than
 1. 4. The imagecapture apparatus according to claim 1, wherein: the image sensorcomprises an analog-to-digital (AD) converter configured to convert thesignals of the plurality of subpixels into digital signals, and thegeneration unit determines the number of tones of the signals of theplurality of subpixels by controlling conversion accuracy of the ADconverter.
 5. A control method of an image capture apparatus having animage sensor in which a plurality of imaging pixels are arranged,wherein each of the plurality of imaging pixels includes a plurality ofsubpixels, and each of the plurality of subpixels receives a light beamthat has passed through a different one of partial pupil areas obtainedby dividing an exit pupil of an imaging optical system into apredetermined number N_(p), wherein N_(p) is an integer greater than 1,the control method comprising: generating, in accordance with a mode,captured image data by combining signals of the plurality of subpixelsfor each of the plurality of imaging pixels, and light field dataconstituted by the signals of the plurality of subpixels, withoutcombining the signals of the plurality of subpixels, wherein, in thegenerating, when generating the light field data, a number of tones ofthe signals of the plurality of subpixels is determined based on thepredetermined number N_(p) so as to be smaller than a number of tones ofthe captured image data.
 6. The control method of an image captureapparatus according to claim 5, wherein in the generating, the modeincludes a first mode in which only the captured image data is obtained,and a second mode in which the captured image data and the light fielddata are obtained, and when generating the light field data in thesecond mode, the number of tones of the signals of the plurality ofsubpixels is determined based on the predetermined number N_(p).